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The  Army  is  rapidly  pursuing  new  reliability  improvement  initiatives  that  support  and  imple¬ 
ment  recent  Defense  Science  Board  and  Department  of  Defense  Reliability  Improvement 
Working  Group  recommendations.  Those  initiatives  are  crucial  and  are  needed  now.  Army 
and  DoD  system  reliability  values  are  on  a  downward  turn.  Even  moderate  degradations 
in  system  reliability,  on  the  order  of  10  percent,  equate  to  billions  of  dollars  in  additional 
costs  over  the  life  cycle  of  a  major  weapon  system.  Even  more  important,  warfighters  deserve 
the  most  reliable  and  rugged  equipment  possible.  The  new  Army  initiatives  will  provide  an  op¬ 
portunity  to  improve  system  reliability,  stop  the  downward  spiral,  and  cost-effectively  implement 
reliability  best  practices. 


The  Army  is  taking  a  multi-pronged  approach  to  improve 
reliability  by  establishing  new  reliability  test  thresholds, 
evaluating  programs  early  using  a  new  reliability  scorecard, 
developing  reliability  growth  tools,  and  increasing  emphasis 
on  early  engineering  analyses  to  positively  affect  designs 
during  the  development  process.  The  Army's  initiatives,  de¬ 
tailed  in  this  article,  can  be  applied  across  DoD. 

Army  Reliability  Policy 

Under  the  new  Army  reliability  policy  approved  in  Decem¬ 
ber  2007,  an  engineering  and  manufacturing  development 
phase  reliability  test  threshold  will  be  established  for  all 
programs  with  a  Joint  Requirements  Oversight  Council  joint 
potential  designator  of  “joint  interest."  The  threshold  values 
will  also  be  incorporated  into  solicitations  for  contracts.  The 
policy— currently  being  added  to  the  next  revisions  of  Army 
Regulation  70-1,  "Army  Acquisition  Policy,"  and  Army  Regu¬ 
lation  73-1,  "Test  and  Evaluation  Policy"— includes  details  for 
establishing  the  system  reliability  threshold.  The  threshold 
will  be  approved  as  part  of  the  test  and  evaluation  master 
plan  and  recorded  in  the  acquisition  program  baseline  at 
Milestone  B.  The  system  will  be  expected  to  meet  or  exceed 
the  reliability  threshold  at  the  end  of  the  first  full-up,  inte¬ 
grated,  system-level  developmental  test  event. 

Achievement  of  the  reliability  threshold  will  be  a  major  focus 
during  design  reviews.  If  a  reliability  threshold  breach  occurs, 
an  in-process  review  led  by  the  Army  Test  and  Evaluation 
Command  will  convene  to  address: 

■  The  program  manager's  planning  and  implementation  of 
corrective  actions  and  associated  impacts 

■  The  Army  Test  and  Evaluation  Command's  assessment 
of  the  corrective  action  plan 

■  Ownership  cost  impacts 

■  System  utility  impact  assessments  from  the  Army  Train¬ 
ing  and  Doctrine  Command. 

The  new  policy  also  highlights  some  of  the  best  commercial 
and  defense  reliability  practices  that  programs  should  use 
to  help  ensure  that  the  system  reliability  requirements  will 
be  met.  The  policy  will  provide  senior  Army  leadership  an 
earlier  warning  for  those  programs  that  are  falling  short  of 
critical  reliability  targets. 

Reliability  Program  Standard 

DoD  worked  closely  with  both  industry  and  the  Govern¬ 
ment  Electronics  and  Information  Technology  Association 
on  the  development  of  a  new  standard:  GEIA-STD-0009, 
"Reliability  Program  Standard  for  Systems  Design,  Develop¬ 
ment,  and  Manufacturing."  DoD  was  motivated  to  initiate 
and  support  the  undertaking  because  many  systems  have 
not  been  achieving  the  required  level  of  reliability  during 
developmental  testing  and  have  been  subsequently  found 
unsuitable  during  initial  operational  test  and  evaluation. 
In  May  2008,  the  Defense  Science  Board  developmental 
test  and  evaluation  task  force  examined  those  issues  and 
concluded  that  a  new  reliability  program  standard— which 


includes  reliability  growth  as  an  integral  part  of  design  and 
development,  and  can  be  readily  cited  in  DoD  contracts— 
was  urgently  needed. 

GEIA-STD-0009  consists  of  the  essential  reliability  pro¬ 
cesses  that  must  be  performed  in  order  to  design,  build, 
and  field  reliable  systems.  GEIA-STD-0009  is,  at  its  core, 
a  reliability  engineering  and  growth  process  that  is  fully 
integrated  with  systems  engineering.  In  order  to  facilitate 
its  use  in  DoD  acquisition  contracts,  enabling  sample  lan¬ 
guage  was  developed;  it  which  can  be  viewed  at  the  Defense 
Acquisition  University's  Acquisition  Community  Connec¬ 
tion  Web  site,  <https://acc.dau.mil/communitybrowser. 
aspx?id=219127&lang=en-us>. 

The  sample  reliability  language  consists  of  four  parts: 

■  Section  C,  Statement  of  Work  Reliability  Language  and 
Tailoring  Instructions.  If  Section  C  of  a  request  for  pro¬ 
posal  contains  a  statement  of  work,  it  is  recommended 
that  this  sample  reliability  language  be  incorporated. 
Embedded  tailoring  guidance  is  included  in  the  sample 
language.  If  Section  C  does  not  contain  a  statement  of 
work,  then  it  is  recommended  that  a  statement  be  in¬ 
cluded  in  the  statement  of  objectives  requiring  that  the 
sample  reliability  language  be  included  in  the  contractor 
statement  of  work. 

■  Section  L,  Proposal  Instructions  Reliability  Language. 
Section  L  of  a  government  contract  lays  out  the  specific 
preparation  requirements  for  submissions. 

■  Section  M,  Evaluation  Factors  for  Award  Reliability  Lan¬ 
guage.  Section  M  relays  the  factors  used  to  determine 
how  the  government  plans  to  compare  each  bid  and 
which  criteria  are  most  important  to  them. 

■  Checklist  for  Evaluating  Reliability  Program  Plans.  The 
checklist  can  be  used  to  evaluate  draft  reliability  pro¬ 
gram  plans  developed  based  on  the  reliability  statement 
of  work  language. 

It  is  also  recommended  that  GEIA-STD-0009  be  explicitly 
cited  in  the  system  specification,  which  is  typically  included 
in  Section  C  of  the  request  for  proposal. 

Reliability  Scorecard 

The  reliability  scorecard  examines  a  supplier's  use  of  reliabil¬ 
ity  best  practices  and  the  supplier's  planned  and  completed 
reliability  tasks.  The  scorecard  can  also  be  used  to  evaluate  a 
given  program's  reliability  progress.  The  scorecard  was  de¬ 
veloped  based,  in  part,  on  reliability  assessment  approaches 
developed  by  the  Institute  of  Electrical  and  Electronics  Engi¬ 
neers,  Raytheon,  Alion,  the  University  of  Maryland,  and  oth¬ 
ers.  The  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 
and  the  Army  Evaluation  Center  expanded  and  refined  the 
individual  assessment  areas  based  on  several  years  of  evalu¬ 
ation  and  reliability  program  experience. 

The  latest  version  of  the  scorecard  allows  quantitative  as¬ 
sessment  across  eight  critical  areas: 
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FROM  OUR  READERS 


Great  Lessons  Learned 

I  would  like  to  praise  Wayne  Turk's  article  “Be 
Willing  to  Make  Changes/'  which  appeared 
in  the  May-June  2009  issue  of  Defense  AT&L 
magazine.  It  is  very  timely  considering  all  that 
is  going  on  in  the  DoD  environment.  Not  only 
was  the  article  concise  and  well-written,  but 
the  author  hit  all  the  critical  points  of  change 
management,  especially  participative  involve¬ 
ment,  explaining  what's  in  it  for  those  impacted 
by  the  change,  creating  a  sense  of  urgency,  and 
using  middle  managers  to  explain  the  new  pro¬ 
cess  and  its  benefits.  Great  idea  for  a  checklist! 

To  quote  Woodrow  Wilson,  “If  you  want  to 
make  enemies,  try  to  change  something."  That 
is  very  true!  Hopefully,  Defense  AT&L  readers 
will  take  advantage  of  Turk's  article  to  select 
the  right  changes  to  make  and  implement 
them  well.  Too  often,  managers  naively  expect 
that  those  impacted  by  the  change  to  embrace 
it  and  understand  it  from  an  “all-hands"  e-mail. 
Successful  change,  as  Turk  points  out  in  his 
article,  takes  a  lot  of  planning  and  work. 

I  also  liked  the  article  “Leaders  as  Circus  Per¬ 
formers,"  by  Fred  Jones,  Doug  McCallum,  and 
Chris  Sargent,  also  appearing  in  the  May-June 
2009  issue.  The  analogy  of  plate-spinners 
(high-level  leaders)  versus  jugglers  (lower- 
level  leaders)  is  a  good  one.  I  also  liked  the 
analysis  of  the  feedback  the  authors  got  from 
previous  surveys.  Too  often,  people  are  asked 
to  respond  to  a  survey,  never  hearing  the  re¬ 
sults  or,  more  important,  never  learning  what 
actions  were  taken  as  a  result  of  the  survey. 

The  authors  also  hit  the  key  reasons  surveys 
are  not  used  more  often  by  leaders:  ignorance, 
fear,  and  skepticism.  Hopefully,  the  article  will 
help  leaders  overcome  those  feelings. 

Al  Kaniss 

Naval  Air  Systems  Command 
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■  Reliability  requirements  and  planning 

■  Training  and  development 

■  Reliability  analysis 

■  Reliability  testing 

■  Supply  chain  management 

■  Failure  tracking  and  reporting 

■  Verification  and  validation 

■  Reliability  improvements. 

Each  element  is  rated  red,  yellow,  or  green  based  on  a  num¬ 
ber  of  questions.  Quantitative  risk  scores  are  provided  for 
each  assessment  area  as  well  as  for  the  overall  system.  This 
scorecard  is  important  for  tracking  the  achievement  of  reli¬ 
ability  requirements  and  rating  the  adequacy  of  the  overall 
reliability  program.  The  scorecard  can  be  accessed  from  the 
Defense  Acquisition  University's  Acquisition  Community 
Connection  Web  site  at  <https://acc.dau.mil/community- 
browser.aspx?id=2104835dang=en-us>. 

New  Reliability  Growth  Tools  and  Test  Bed 

AMSAA  has  produced  several  new  reliability  growth  mod¬ 
els.  One  such  model  is  the  planning  model  based  on  pro¬ 
jection  methodology  (PM2).  Many  times,  analysts  will  pro¬ 
duce  system-level  reliability  growth  planning  curves  that, 
at  first  glance,  appear  reasonable  and  achieve  the  desired 
goals  with  the  given  set  of  input  parameters.  However,  those 
curves  often  do  not  allow  for  the  impacts  associated  with 
schedule,  testing,  refurbishment,  and  block  updates.  By  not 
accounting  for  those  very  real  constraints,  the  system-level 
reliability  growth  planning  curves  can  portray  an  overly  opti¬ 
mistic  and  unrealistic  program  for  achieving  the  system-level 
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reliability.  Those  constraints  could  be  addressed  using  older 
reliability  growth  methodology,  but  not  in  a  systematic  way. 

Planning  models  can  be  made  more  realistic  by  incorporating 
some  of  the  methodology  used  for  reliability  growth  projec¬ 
tion,  as  demonstrated  by  the  P!\A2  model.  The  projection 
methodologies  account  for  key  engineering  and  schedule 
decisions  such  as  fix  effectiveness  factor  levels,  manage¬ 
ment  strategy,  delays  for  incorporating  fixes,  and  refurbish¬ 
ment  period  scheduling. 

The  P!\A2  model  starts  by  determining  the  reliability  test¬ 
ing  operating  hours,  which  are  officially  scored,  as  a  func¬ 
tion  of  calendar  time  for  each  individual  system  included  in 
reliability  growth  testing.  Then  for  each  platform  calendar 
schedule,  the  blocks  of  time  where  corrective  actions  are 
implemented  (i.e.,  refurbishment  periods)  are  inserted. 
Next,  an  estimate  for  the  average  time  between  the  occur¬ 
rence  of  a  new  problem  failure  mode  and  when  a  corrective 
action  can  be  inserted  is  applied.  The  user  then  develops  an 
idealized  growth  curve  using  projection  methodology.  The 
user  specifies  the  initial  mean  time  between  failure,  the  goal 
mean  time  between  failure,  the  planned  value  of  the  average 
fix  effectiveness  factor,  the  management  strategy,  and  the 
allocated  test  time.  Those  five  values  define  the  idealized 
growth  curve.  The  idealized  curve  gives  the  expected  mean 
time  between  failure  as  a  function  of  test  time;  and  offers 
a  number  of  highly  useful  metrics  that  provide  the  program 
manager  and  other  members  of  the  acquisition  community 
with  a  valuable  means  to  assess  the  reliability  program,  test¬ 
ing  program,  number  of  assets  available,  and  the  availability 
of  engineering  resources  to  maximize  the  chances  of  produc¬ 
ing  a  highly  reliable  and  cost-effective  system. 

The  AMSAA  PM2  reliability  growth  planning  curve  sets  a 
much  better  expectation  for  what  reliability  values  should 
be  achieved  as  part  of  the  system  development  process. 
Often  with  idealized  curves,  the  reliability  values  are  overly 
optimistic.  In  many  programs,  a  large  portion  of  the  testing 
hours  actually  occurs  towards  the  end  of  the  program  or 
immediately  before  a  major  milestone.  The  idealized  curve 
often  shows  that  the  desired  reliability  is  achieved  by  the 
milestone.  However,  when  actual  schedule  and  corrective 
constraints  are  placed  on  the  planned  reliability  growth 
curve,  engineers  and  management  can  see  where  the  real 
bottlenecks  are  and  better  allocate  their  test  time,  engineer¬ 
ing  activities,  refurbishment  periods,  and  test  assets  in  order 
to  meet  the  reliability  goal  with  minimal  risk.  The  model  has 
recently  been  released  in  a  Microsoft®  Excel  spreadsheet 
format  and  is  available  to  U.S.  government  employees  and 
defense  contractors,  who  can  request  access  to  the  model 
by  e-mailing  apgr-amsa-reltools(a)conus. army.mil. 

Another  supporting  advance  is  the  new  reliability  simulation 
test  bed,  which  examines  the  potential  reliability  growth  of 
complex  systems.  The  goal  of  the  test  bed  is  to  examine 
the  impacts  of  various  reliability  growth  strategies  on  the 


overall  system  reliability  and  the  accuracy  of  associated 
statistical  model  reliability  assessments.  The  simulations 
are  conducted  by  making  random  draws  of  initial  mode  fail¬ 
ure  rates  from  several  parent  populations.  The  simulation  is 
unique  in  comparison  to  others  of  this  type  in  that  it  allows 
for  flexibility  in  implementing  corrective  actions.  The  correc¬ 
tive  actions  may  be  implemented  during  the  test  phase  of 
the  system  or  delayed  until  the  end  of  the  overall  test  period. 
When  implemented  during  the  test  phase,  the  corrective 
actions  are  implemented  either  through  built-in  corrective 
action  periods  within  the  overall  test  phase  or  during  the  test 
periods  themselves. 

In  the  near  term,  existing  Army  reliability  growth  models  are 
to  be  converted  to  an  Excel-based  format  to  allow  more  DoD 
personnel  and  contractors  to  easily  implement  the  latest  in 
reliability  growth  modeling  without  cost. 

Physics  of  Failure 

The  Army  continues  to  successfully  apply  physics  of  failure 
modeling  to  a  wide  variety  of  vehicles  and  electronics  sys¬ 
tems.  PoF  is  a  science-based  approach  to  reliability  that  uses 
modeling  and  simulation  to  design  reliability  into  a  prod¬ 
uct,  perform  reliability  assessments,  and  focus  reliability 
tests  and  screens  where  they  will  be  the  most  effective  and 
productive.  The  PoF  approach  involves  modeling  the  root 
causes  of  failure— often  called  failure  mechanisms— such 
as  fatigue,  fracture,  wear,  and  corrosion.  The  basis  of  PoF 
is  that  it  is  not  only  important  to  understand  how  things 
work  but  also,  equally  important,  to  understand  how  things 
fail.  Computer-aided  design  tools  have  been  developed  to 
address  various  loads,  stresses,  failure  mechanisms,  and 
failure  sites. 

PoF  modeling  has  been  critical  for  systems  currently  sup¬ 
porting  Operation  Iraqi  Freedom  and  Operation  Enduring 
Freedom.  For  example,  the  PoF  modeling  provided  a  quick 
and  efficient  way  to  mitigate  the  weight  impacts  of  new 
armor  packages  and  maintain  system  reliability  for  warf¬ 
ighters. 

Increasing  Weapons  System  Reliability 

The  initiatives  discussed  in  this  article  are  just  some  of  the 
ways  that  the  Army  is  working  to  further  increase  weapons 
system  reliability.  These  initiatives  are  cost-effective  and  will 
provide  significant  returns  on  investment— and  even  greater 
benefits  for  our  warfighters. 

U.S.  government  personnel  and  their  DoD  contractors  in¬ 
terested  in  obtaining,  at  no  cost,  reliability  growth  models, 
the  reliability  scorecard,  or  associated  training  can  send  an 
e-mail  request  to  apgr-amsa-reltools@conus.army.mil. 


The  authors  welcome  comments  and  questions  and  can  be 
contacted  at  apgr-amsa-reltools@conus.army.mil. 
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